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SUMMARY  AND  HIGHLIGHTS 


What  is  the  relationship  between  the  number  of  critical  incidents 
recalled  and  the  number  of  months  before  the  time  of  reporting  that  the 
incidents  occurred? 

To  answer  that  question  we  tabulated  critical  incidents  reported  as 
having  occurred  one  to  twelve  months  before  the  collection  date:  approx- 
imately 800  from  Personnel  Officers  GS  12  through  15,  700  from  Data  Pro- 
cessors GS  5 through  15,  and  1,000  from  Plant  Quarantine  Inspectors  GS  5 
through  13.  These  included  incidents  of  both  effective  and  ineffective 
behaviors  in  the  general  areas  of  f;technical  competence'-'  and  ''working 
with  others. r Typically,  between  six  and  seven  incidents  were  obtained 
from  each  contributor. 

When  the  number  of  incidents  reported  for  the  immediately  preceding 
six  month  period  was  compared  with  the  number  that  occurred  seven  to 
twelve  months  ago,  in  each  group,  the  superiority  of  yield  for  the  more 
recent  period  was  156%  to  4007o. 

These  results  show  why  biases,  irritations,  and  frustrations  are 


induced  by  the  typical  performance  appraisal,  that  requires  the  supervi- 
sor to  try  to  remember  for  all  of  his  subordinates,  all  of  their  impor- 
tant performances,  for  all  of  a year.  The  use  of  Performance  Records  to 
make  note  of  critical  incidents  as  they  occur,  such  as  are  being  developed 
for  USDA  occupations,  help  to  ameliorate  those  conditions. 


Albert  S.  Glickman 

Chief,  Personnel  Research  Staff 


Ill 


TABLE  OF  CONTENTS 

Page 

Introduction  1 

Collection  of  Incidents  .......  2 

Description  of  the  Groups  3 

Personnel  3 

Data  Processors 3 

Plant  Quarantine  Inspectors  4 

Analysis  of  the  Incidents  4 

Discussion  of  Results  and  Recommendations  7 

Appendix  A 9 

Appendix  B 11 

LIST  OF  EXHIBITS 

Page 

1.  Percent  of  Incidents  Recalled  Compared  With  the  Number  of 

Months  Elapsed  Since  the  Incident  Occurred  5 

2.  Graphic  Display  of  Percent  of  Incidents  Recalled  Compared 
With  the  Number  of  Months  Elapsed  Since  the  Incident 

Occurred . . . 6 

3.  Number  of  Months  Elapsed  Since  the  Incident  Occurred  12 


1 


RECALL  OF  CRITICAL  INCIDENTS 

Introduction 

During  a work  day  a person  will  perforin  many  different  acts  in  his 
job.  Some  acts  will  advance  accomplishment  of  the  job;  other  acts  will 
have  the  opposite  effect.  Specific  examples  of  behaviors  that  contri- 
bute significantly  to  effective  or  ineffective  accomplishment  of  a job 
have  been  called  critical  incidents.^ 

A method  of  determining  the  critical  requirements  of  a job  or 
occupation  is  to  collect  reports  of  critical  incidents  from  people 
directly  involved  in  the  work.  Employees,  supervisors,  or  others  who 
are  in  a position  to  make  direct  observations  are  asked  to  cite  specific 
occurrences  that  led  to  effective  and  ineffective  accomplishment  of  the 
job.  To  encourage  adequate  coverage  and  representativeness,  respondents 
may  be  requested  to  cite  examples  for  particular  categories  such  as 
'technical  competence*  and  ‘'working  with  others.*  For  each  incident, 
the  respondent  tells  what  the  individual  concerned  was  doing  at  the  time, 
what  happened,  and  what  the  consequences  were.  Information  on  the  nature 
and  level  of  the  jobs  held  by  the  persons  involved  and  how  long  ago  the 
incident  occurred  is  also  requested  when  desired  for  analysis.  The 
anonymity  of  all  persons  involved  in  the  incidents  as  well  as  the  respond- 
ents is  maintained. 

This  report  presents  the  results  of  an  inquiry  into  the  effects  of 
the  passage  of  time  upon  the  recall  of  critical  incidents,  viz.  the 


Flanagan,  John  C.  The  Critical  Incident  Technique.  Psychological 
Bulletin,  vol.  51,  no.  4,  July  1954,  pp.  327-358. 
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relationship  between  the  number  of  critical  incidents  reported  and  the 
number  of  months  since  the  incidents  occurred. 

Collection  of  Incidents 

The  Personnel  Research  Staff  (PRS)  and  representatives  of  the 
following  occupational  groups  are  currently  engaged  in  critical  inci- 
dent studies:  (1)  General  Foresters;  (2)  Plant  Quarantine  Inspectors; 
(3)  Soil  Conservationists;  (4)  Data  Processors;  and  (5)  Personnel  people 

Representative  samples  of  respondents  from  each  of  these  occupa- 
tional groups  were  asked  to  give  two  effective  and  two  ineffective  inci- 
dents in  each  of  two  areas,  "job  or  technical  competence'5  and  "working 
with  others' ; eight  incidents  from  each  respondent.  Sometimes  a respond 
ent  could  not  think  of  an  example  for  one  of  the  categories,  and  hence 
the  average  yield  per  respondent  was  between  six  and  seven  incidents. 
Additional  information  was  requested  on  the  job  duties  of  the  people 
involved  in  the  incidents  as  well  as  of  the  respondents. 

Respondents  were  told  that  the  incidents  and  background  information 
were  being  collected  for  research  purposes.  Almost  all  incidents  were 
collected  during  1963.  Three  of  these  occupational  groups,  Plant  Quar- 
antine Inspectors,  Data  Processors,  and  Personnel,  requested  information 
as  to  the  number  of  months  before  the  time  of  reporting  that  the  inci- 
dent occurred.  The  remainder  of  this  report  analyzes  the  relationship 
between  the  number  of  incidents  recalled  and  the  number  of  months  before 
the  time  of  reporting  that  the  incidents  occurred,  for  these  three  occu- 


pational groups. 
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Description  of  the  Groups 
Personnel 

The  Personnel  group  included  GS  12s  through  15s  in  the  following 


series : 

201  - 

Personnel  Administration 

212  - 

Placement 

221  - 

Position  Classification 

223  - 

Salary  and  Wage  Administration 

230  - 

Employee-Management  Relations 

235  - 

Employee  Development 

A near  100  percent  sample  of  people  in  the  above  classifications 
was  obtained.  Although  the  Personnel  group  is  most  heavily  concentrated 
in  the  Washington,  D.  C.  area,  some  are  in  each  region  of  the  United 
States.  Approximately  800  incidents  were  collected  by  a representative 
of  the  workgroup  of  agency  personnel  officers  participating  in  this  project 
Data  Processors 

Included  in  this  group  were  GS  5s  through  15s  in  the  following  series: 


330  - 

Digital  Computer  Systems  Administrator 

331  - 

Digital  Computer  Programming 

332  - 

Digital  Computer  Systems  Operation 

334  - 

Digital  Computer  Systems  Analysis 

359  - 

Electric  Accounting  Machine  Operating 

362  - 

Electric  Accounting  Machine  Project  Planning 

A near  100  percent  sample  of  people  in  the  above  classifications  was 
obtained.  Approximately  700  incidents  were  collected  by  either  personnel 
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people  or  by  persons  designated  to  serve  as  liaison  between  the  Data 
Processing  workgroup  and  the  Data  Processors. 

Plant  Quarantine  Inspectors 

Included  in  this  group  were  GS  5s  through  GS  13s  in  Plant  Quaran- 
tine Inspection  (GS-436).  Approximately  50  percent  ©f  people  in  this 
classification  were  included  in  this  sample.  Plant  Quarantine  Inspectors 
are  stationed  at  geographically  scattered  points  of  entry  into  the  United 
States.  Some  incidents  were  collected  by  personnel  officers  using  a group 
interview.  More  remote  points  of  entry  were  reached  by  mailing  out  inci- 
dent forms  and  instructions.  A greater  proportion  ©f  the  incidents  were 
collected  from  inspectors  at'  the  major  ports  such  as  New  York  and  New 
Orleans.  A total  of  about  1,000  incidents  were  collected. 

Analysis  of  the  Incidents 

Each  respondent  was  asked  to  think  back  over  the  past  year  and 
recall  occasions  when  he  saw  an  employee  do  things  particularly  well  or 
poorly.  For  each  occupational  group  a tabulation  was  made  of  the  number 
of  months  ago  that  the  incident  occurred.  From  what  is  known  of  human 
memory  one  would  expect  that  the  more  recent  incidents  would  be  remembered 
more  readily  than  incidents  occurring  at  earlier  times. 

A coding  scheme  had  to  be  devised  for  tabulating  those  responses 
that  were  not  numerical  (e.g.  some  respondents  indicated  that  an  incident 
occurred  a few  or  several  months  ago).  This  coding  scheme  is  given  in 
Appendix  A • 

Some  respondents  reported  incidents  that  occurred  longer  than  a 
year  ago.  Incidents  were  tabulated  as  having  occurred  for  each  of  the 
preceding  twelve  months,  plus  two  additional  categories  for  more  than 
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twelve  months  and  for  no  'response,'  These  tabulations  of  effective, 
ineffective,  and  total  incidents  for  each  group  are  listed  in  Appendix 
B.  Since  there  was  no  discernible  systematic  difference  in  the  recall 
of  effective  and  ineffective  incidents,  'total  incidents  recalled'  were 
counted  for  each  of  the  twelve  months  prior  to  reporting. 

Inspection  of  the  tabulations  in  Appendix  B shows  that  the  respond- 
ents had  a marked  preference  for  certain  numbers  when  reporting  the 
number  of  months  ago  that  the  incident  occurred.  The  numbers  one,  two, 
three,  six  and  twelve  were  most  frequently  used.  In  order  to  iron  out 
the  effects  of  this  number  preference,  comparisons  were  made  of  the  per- 
cent of  incidents  reported  as  occurring  during  the  more  recent  six  month 
period  versus  the  earlier  half  year.  These  numerical  comparisons  are 
given  in  Exhibit  1 and  are  displayed  graphically  in  Exhibit  2. 

Exhibit  1 

Percent  of  Incidents  Recalled  Compared  With  the  Number 
of  Months  Elapsed  Since  the  Incident  Occurred 


Six  or  Less 

Seven  Through 
Twelve 

Ratio  of  More  Recent 
to  Earlier  Period 

Personnel 

61 

39 

1.56 

Plant  Quarantine 
Inspectors 

66 

34 

1.94 

Data  Processors 

80 

20 

4.00 

NUMBER  OF  MONTHS  ELAPSED  SINCE  INCIDENT  OCCURRED 
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PERCENT  OF  INCIDENTS  RECALLED 


GRAPHIC  DISPLAY  OF  PERCENT  OF  INCIDENTS  RECALLED 
COMPARED  WITH  THE  NUMBER  OF  MONTHS  ELAPSED 
SINCE  THE  INCIDENT  OCCURRED 
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Inspection  of  Exhibits  1 and  2 indicates  that  the  greater  number 
of  incidents  are  recalled  from  the  six  month  period  immediately  pre- 
ceding the  date  of  reporting.  For  Personnel  people  approximately  one 
and  one-half  times  as  many  incidents  were  recalled  from  the  immediately 
preceding  six  month  period  as  from  the  earlier  six  month  period.  For 
the  Plant  Quarantine  Inspectors  almost  twice  as  many  incidents  were 
recalled  from  the  more  recent  six  month  period*  For  the  Data  Processors, 
four  times  as  many  incidents  were  recalled  from  the  immediately  preceding 
six  month  period. 

Discussion  of  Results  and  Recommenda t i ons 

It  is  consistently  clear  that  over  a one  year  reporting  period  many 
more  incidents  are  recalled  from  the  six  month  period  immediately  pre- 
ceding the  reporting  date  than  from  the  earlier  six  month  period.  These 

2 

results  are  consonant  with  the  results  obtained  in  an  industrial  setting. 
This  study  showed  that  when  incidents  were  recorded  on  a weekly  basis, 
they  were  only  half  the  number  that  were  recorded  on  a daily  basis,  and 
at  the  end  of  a two-week  period  only  one-fifth  as  many  incidents  were 
recorded. 

Imagine  now  a supervisor  at  appraisal  time  who  must  take  time  out 
from  his  work  to  try  to  remember  what  all  of  his  supervisees  have  done 
during  the  past  year.  It  is  not  surprising  that  many  supervisors  not 
only  find  this  procedure  frustrating  and  irritating  but  feel  that  they 
cannot  formulate  useful  judgments.  Small  wonder  then  that  many 


Flanagan,  J.  C.  6:  Burns,  R.  K.  'The  Employee  Performance  Record*  . 
In  Whisler,  L.  L.  & Harper,  S.  F.  Performance  Appraisal : Research  and 
Practice.  New  York:  Holt,  Rinehart  & Winston,  1962. 


supervisors  develop  a negative  attitude  towards  the  entire  appraisal 
procedure  and  tend  to  shy  away  from  it  rather  than  try  to  make  it  a 
positive  contribution  to  the  supervisor-supervisee  relationship. 

A report  dealing  with  the  steps  involved  in  the  collection  and 
analysis  of  critical  incidents  as  well  as  their  utilization  in  a Per- 
formance Record  system,  in  which  incidents  are  recorded  as  they  occur 
will  be  forthcoming  in  this  Personnel  Research  Series. 


10 


Coding  Scheme  for  Non-Numerical  Responses  to  the  Question 
'About  How  Many  Months  Ago  Did  the  Incident  Occur?' 


1.  When  more  than  one  alternative  is  given,  e.g. 
number  representing  the  longest  period  of  time,  i.e.  6 

2.  Score  several  as  six,  a few  as  three. 

3.  Whenever  a period  of  time  is  specified  try  to 
for  that  period.  Thus,  score: 

a few  weeks 


dai  ly 

continuously 

always 

all  the  time 


as  one  month 


5 or  6,  choose  the 
in  this  example. 

score  the  response 


as  no  response 


recurring 


' 
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